Revise network.py to support per-run and overall test criteria (New)#2378
Open
Revise network.py to support per-run and overall test criteria (New)#2378
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2378 +/- ##
==========================================
- Coverage 57.66% 57.64% -0.02%
==========================================
Files 465 465
Lines 47166 47179 +13
Branches 8398 8400 +2
==========================================
Hits 27198 27198
- Misses 19088 19098 +10
- Partials 880 883 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
rodwsmith
force-pushed
the
change-network-defaults
branch
from
4681619 to
d3b4094
Compare
rodwsmith
force-pushed
the
change-network-defaults
branch
from
d3b4094 to
c6faf90
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title: Revise network.py to support per-run and overall test criteria (New)
Description
This PR adds a new command-line parameter (
--overall-fail-threshold) to thenetwork.pyscript, enabling setting separate per-run and overall failure thresholds. That is, previously, if--fail-thresholdwas set to, say, 90, and if--num_runswas set to, say, 4, then this script would cause a re-run if any individual run fell below 90%, and if the timeout period passed and any run was under 90%, then the script would return an error code and the test would fail. This change permits setting a separate criterion for the overall failure threshold. That is, if--fail-thresholdis set to 80 and--overall-fail-thresholdis set to 90, then with four runs, results on individual runs could fall as low as 80% and not trigger a re-run, so long as the final average is above 90% -- so, for example, if the individual results were 92%, 93%, 89%, and 93%, the test would pass without re-running. The new--overall-fail-thresholdparameter is optional; if it's missing, the value defaults to the same value set by--fail-threshold. Thus, there is no change in the script's behavior if the new option is not specified.This PR also adjusts the
providers/base/unit/ethernet/jobs.pxufile so that the new option is set to 90 for theethernet/sru_iperf3_stress_device-{__index__}_{interface},ethernet/multi_iperf3_nic_device{__index__}_{interface}, andethernet/multi_iperf3_nic_underspeed_device{__index__}_{interface}tests, used exclusively in server certification. This change satisfies the Jira task noted below, as discussed by Jeff Lane and Rod Smith to create stricter pass criteria (previously set to 80) without creating undo problems because of occasional sub-runs that produce sub-90% results.Resolved issues
(https://warthogs.atlassian.net/browse/SERVCERT-1893)
Documentation
No changes are required to Checkbox documentation. Changes will be needed to some of the Server Certification partner documentation. These changes will be implemented as part of the updates for the Ubuntu 26.04 release.
Tests
Testing was done by running
network.pymanually, with commands like the following:This was a shorter run than the usual test (which takes an hour to complete), for ease of testing. The
--fail-thresholdand--overall-fail-thresholdvalues were adjusted so that the machine's naturally tested speeds passed or failed and observing the results, which matched expectations; for instance, when speeds were about 94%, setting--fail-thresholdto 80 and--overall-fail-thresholdto 95 caused no re-runs of the individual runs, but the overall test failed, as expected. When--overall-fail-thresholdwas reduced to 90, the test passed.In addition, the complete set of tests was run in Checkbox, having replaced both files. The results are here:
https://certification.canonical.com/hardware/201409-15506/submission/476574/
The relevant run was:
https://certification.canonical.com/hardware/201409-15506/submission/476574/test/74015/result/56526890/
These results are not notably different from runs with the previous version of the script; but there is a new line that provides the overall result at the end of the test output.